The present invention relates to a monitoring apparatus provided with a TV camera and more particularly to a method for tracking an object and an apparatus for tracking and monitoring an object which are both arranged to automatically detect one or more objects entering or intruding in a field of view of a TV camera from a video signal sent from the TV camera and to track movement of the detected object(s) or magnify the object(s) on a TV monitor.
A manually operated picture monitoring apparatus arranged to use a TV camera has been prevailing in the past. In recent years, however, in place of monitoring the object on the monitor by a person, the type of monitoring system has been demanded which automatically detects a moving object such as a person or a vehicle entering a monitoring field of view from a video signal and display the object in magnified representation or give a certain notice or an alarm.
To detect a moving object or an entering object, a conventional system has been arranged to compare an input video signal obtained from the TV camera and a reference background video signal, that is, a video signal that does not contain an object to be detected, derive a difference of a brightness value at each pixel between both of the video signals, and detect an area with a large difference as an object. This system is called the difference method, which has been widely used. The application of this system is described in U.S. patent application Ser. No. 08/646,018 filed on May 7, 1996 (corresponding to the European Application No.963033030 filed on May 13, 1996). Another application of the tracking system with the difference method is discussed in the thesis entitled xe2x80x9cA procedure for the Analysis of Skiers"" Movement with Image Processing Techniquesxe2x80x9d, Vol. 43, No. 12, pp. 1370 to 1374 of xe2x80x9cProceedings of TV Societyxe2x80x9d issued on December, 1989.
One of plausible methods arranged to automatically detect an entering object from a video signal obtained from a TV camera, track the entering object and magnify it on a video monitor screen for the purpose of facilitating the monitor of the entering object is to detect the entering object by successively switching the video signals outputted from a plurality of TV cameras. However, this method disadvantageously involves many TV cameras and a video signal switcher and a controller therefor so that this method makes the overall system larger in scale.
For decreasing the necessary number of TV cameras for the purpose of reducing the system scale, the system is required to widen a field of view of an imaging lens provided in each TV camera. This system however makes the entering object appear on the monitor in a small size. For example, if the entering object is a person, this system may have difficulty in discriminating the lineaments of the entering person.
Further, when an object having a luminance value close to the background picture is to be detected by the difference method, a part of the object may be lost undetected or the object may be split into a plurality of pieces, resulting in detecting the object as a plurality of objects. If one object within a field of view is detected as a plurality of objects, therefore, the method for tracking an object using the difference method may determine that the same number of objects as the detected member of the objects exist and track the plurality of the objects though in actuality just one object exists within the field of view. Further, the tracking method using the difference method detects objects that are seemingly connected together and looks as one object as one object. In a case that, therefore, a plurality of objects exist within the field of view, if these objects are connected together at a certain point in time, this type of method determines that there exists only one object at the time and hence track them as one object though in actuality two or more objects exist within the field of view. Moreover, since the tracking method using the difference method involves no comparison of shapes of a picture nor no comparison of picture patterns, if two or more objects are located within the field of view, this method can not reliably guarantee that the same object is being tracked.
It is an object of the present invention to provide a picture processing system for tracking an entering object by automatically detecting the entering object and magnifying the detected entering object, for allowing an observer to more easily make sure of the entering object on a video monitor screen.
In carrying out the object, a picture processing system for tracking an entering object according to an aspect of the invention includes a camera or a TV camera with an imaging lens of a wide field for outputting a video signal used for detecting the entering object, another camera or TV camera with a zoom lens for outputting a video signal used for monitoring the picture as tracking the entering object, the camera being mounted on an electric pan and tilt head or pan head, and a picture processing device for processing and operating a video signal inputted from the camera provided with the lens of the wide field and outputting a rotation control signal for rotating the electric pan and tilt head by a necessary amount of angle, wherein the object entering the predetermined monitoring area is automatically tracked and picked up. Here, the camera or TV camera provided with the imaging lens of a wide field is a camera or a TV camera which is capable of imaging the whole of the picture within the object tracking and monitoring area. The whole of the object tracking/monitoring area is referred to as the wide field or field of view.
A system for processing a tracking picture of an entering object according to another aspect of the invention includes a camera or a TV camera with a lens of a wide field for outputting a video signal used for detecting the entering object, another camera or TV camera with a zoom lens for outputting a video signal used for monitoring a magnified picture as tracking the entering object, the camera being mounted on an electric pan and tilt head, a picture processing device for processing and operating the video signal sent from the camera with the lens of the wide field and outputting a rotation control signal for rotating the electric pan and tilt head by a necessary amount of angle and a zooming control signal for zooming the zoom lens by a necessary amount, wherein the object entering the predetermined monitoring area is automatically tracked and picked up.
Preferably, the picture processing device in the system includes a picture memory for storing a video signal sent from the camera with the lens of the wide field and a processed signal of the video signal, a CPU (central processing unit) for carrying out processing, operation, control and management performed in the picture processing device, a program memory for storing a program for operating the CPU, a working memory for storing data for analyzing data obtained by processing a locational coordinate and a movement history of the detected entering object, and a pan and tilt head control unit for outputting the rotation control signal for rotating the electric pan and tilt head by a necessary amount of angle and the zooming control signal for zooming the zoom lens by a necessary amount based on the operated data.
Preferably, the picture memory provided in the picture processing device includes an input picture memory for storing an inputted video signal, a background picture memory for storing a background video signal used for detecting the entering object, two working memories for storing the background video signal and the input video signal for use in carrying out an operation of detecting a difference between the pictures, a picture memory for storing an inter-picture operation result obtained by detecting and operating a difference between the background picture and the input picture, and a binary-coded picture memory for storing a binary-coded video signal obtained by operating the detected picture difference.
A system for processing a tracking picture of an entering object according to an embodiment of the invention is arranged to process picture information inputted from a camera or a TV camera having a lens of a wide field into a picture processing device and detect the entering object, and rotate an electric pan and tilt head having a camera or a TV camera with the zoom lens mounted thereon toward the entering object from the location information and the size information of the detected entering object as well as control the zooming of the zoom lens, for allowing an observer of the monitor to easily determine the entering object.
It is a further object of the present invention to provide a highly reliable apparatus for tracking and monitoring an object which is arranged to determine if splitting of one object has occurred or if a plurality of objects exist, although they are merged or connected as one object.
It is a yet further object of the present invention to provide an apparatus for tracking and monitoring an object which is arranged to infer a precise location of one object even if splitting has occurred in detecting the one object and infer a precise location of each of a plurality of objects even if merging or connection has occurred in detecting the plurality of objects so as to make it possible to detect a tracking path of each object.
It is another object of the present invention to provide an apparatus for tracking and monitoring an object which is arranged to correct a tracking path of each object from the detecting state before and after the time of the detection and to precisely track the object if splitting has occurred in defecting one object or merging or connection has occurred in detecting a plurality of objects.
It is another object of the present invention to provide an apparatus for tracking and monitoring an object which is arranged to guarantee that the same object is being tracked even if two or more objects are located within a target field of view.
In achieving the foregoing objects, according to another aspect of the invention, a change in the number of detected objects is detected and splitting or connection of the object or objects is detected from a change in detecting state.
According to still another aspect of the invention, the method for tracking and monitoring an object includes the steps of detecting an object contained in an input video signal, detecting the change in the number of detected objects other continuous frames, classifying a change of a detecting state of the objects from the detected change in the number of the detected objects, and determining if the detected objects are split from one object or the detected one object is made up of plural objects connected together based on the classified state change of the objects.
An object tracking and monitoring method according to an embodiment of the invention includes the steps of creating connecting information for representing a locational change of an object detected at each time and a change in a detected area from the change of the detecting state and inferring a trace of the object by estimating the locational change of the object within a variable length of time, for realizing more precise object tracking by tracking the detected object and estimating the locational change of the detected object within a long length of time.
According to another aspect of the invention, an object tracking and monitoring method includes the steps of determining if splitting of one object occurs when the object is detected from the aforementioned connecting information and correcting plural areas of the split parts as one area if it is determined that splitting has occurred in the object, whereby the split parts are corrected as one object even when one object is detected as a plurality of objects due to splitting which has temporally occurred in detecting the object.
According to another aspect of the invention, an object tracking and monitoring method includes the steps of determining if merging or connection of a plurality of objects into one object has occurred in object detection from the aforementioned connecting information and correcting the connection of the plural objects from the connecting information before and after the detection if the object is determined to be connected, whereby the plural objects can be detected as plural objects even if they are temporarily detected as one object.
According to another aspect of the invention, an object tracking and monitoring method includes the recursive determining step of determining if the split parts are corrected as one object at the aforementioned step of correcting the split parts, for recursively correcting the split parts as one object. At this step, if one object is detected as the split parts, those parts are more precisely corrected as one object.
According to another aspect of the invention, an object tracking and monitoring method includes the recursive determining step of determining if one object is corrected as the connection of plural objects at the aforementioned step of correcting the connection, for recursively correcting the connected objects as individual objects. At this step, if the connected objects are detected as one object, the object is more precisely corrected as individual objects.
According to another aspect of the invention, an object tracking and monitoring method includes the steps of keeping also as the connecting information the pixel pattern of the detected area together with the location of the detected object and the area of the detected area given at the aforementioned step of creating the connecting information, matching the pixel pattern of the detected area to the pixel pattern kept at the step of determining the connection, for the purposes of determining if the object is a connected one, tracking the detected object, and guaranteeing that the same object is being tracked if plural objects are located in the field of view.
According to another aspect of the invention, an object tracking and monitoring apparatus includes a camera or a TV camera for imaging a field of view to be monitored, a picture input interface for converting a signal from the camera into a video signal, and a processing unit including at least a CPU, a picture memory, a working memory and a program memory, for processing the video signal. The apparatus executes the operations of detecting a difference of a luminance value at each pixel between an input video signal from the camera and a reference background video signal with no object to be detected; detecting the region with a large difference as an object; classifying the change of the detecting state from a change in the number of the detected objects contained over the continuous input video signals; storing as the connecting information a location of the detected object at each time, an area of the detected region and a picture pattern of the detected object, wherein the location, the region and the picture pattern are derived from the change of the detecting state; correcting the detected location of an object if splitting has occurred in detecting the object and correcting the trace of the objects if merging or connection of the objects has occurred in detecting the objects.